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Executive  Summary 


Introduction.  This  report  is  one  in  a  series  of  reports  that  the  Inspector  General,  DoD, 
is  issuing  in  accordance  with  an  informal  partnership  with  the  Chief  Information 
Officer,  DoD,  to  monitor  DoD  efforts  in  addressing  the  year  2000  computer  challenge. 
For  a  lik  of  audit  projects  addressing  the  issue,  see  the  year  2000  web  pages  on  the 
IGnet  at  http://www.ignet.gov/. 

Objective.  The  overall  audit  objective  was  to  evaluate  the  effectiveness  of  Ae  planned 
end-to-end  testing  in  the  DoD  transportation  pay  area.  Specifically,  we  reviewed  the 
Defense  Finance  and  Accounting  Service  Year  2000  End-to-End  Master  Plan  revision 
2.3,  dated  June  2l,  1999,  and  the  plans  for  conducting  end-to-end  testing  within  the 
transportation  pay  process.  The  purpose  of  end-to-end^  testing  is  to  verify  that  the  set  of 
interrelated  systems  supporting  an  organizational  function,  such  as  transportation  pay, 
interoperates  as  intended  in  an  operational  environment. 

Results.  The  Defense  Finance  and  Accounting  Service  made  significant  improvements 
in  end-to-end  testing  of  the  transportation  pay  systems.  The  Defense  Finance  and 
Accounting  Service  plan  for  end-to-end  testing  of  the  transportation  pay  business 
process  was  to  use  the  system-level  testing  as  end-to-end  testing.  However,  at  the  time 
of  the  system-level  testing,  one  of  the  systems  within  the  thread  was  not  certified  as 
year-2000  compliant.  Since  that  time,  the  uncertified  system  has  been  cerUfied,  and  the 
interface  between  the  previously  uncertified  system  and  the  transportation  pay 
accounting  system  was  successfully  re-tested.  In  addition,  tests  were  performed 
successfully  with  the  Federal  Reserve  Bank  on  September  15,  1999.  The  transportation 
pay  systems  should  continue  to  operate  as  intended  in  the  year  2000,  based  on  the  tests 
performed.  See  Audit  Results  for  additional  details. 

Management  Response  to  Inspector  General,  DoD,  Memorandum.  Management 
actions  were  responsive  to  suggestions  made  during  the  review  of  end-to-end  tests.  In 
addition,  the  Defense  Finance  and  Accounting  Service  took  actions  based  on  a 
memorandum  that  we  issued  on  July  9,  1999,  and  contracted  with  the  Joint 
Interoperability  Test  Command  to  review  system-level  tests  to  see  whether  they  met  the 
intent  of  end-to-end  testing.  Management  also  promptly  implemented  Joint 
Interoperability  Test  Command  recommendations  to  reduce  risks. 

Mangement  Comments.  We  provided  a  draft  of  this  report  on  September  17, 1999. 
Because  the  draft  report  contained  no  recommendations,  written  coimnents  were  not 
required,  and  none  were  received.  Therefore,  we  are  publishing  this  report  in  final 
form. 
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Background 


Addressing  the  Year  2000  Computing  Challenge.  The  Defense  Finance  and 
Accounting  Service  (DFAS)  is  the  principal  agency  responsible  for  DoD 
accounting  and  finance  processes.  DFAS  is  also  responsible  for  disbursing 
payments  by  U.S.  Treasury  check,  by  cash,  or  by  electronically  transferring 
funds.  In  FY  1998,  DFAS  processed  a  monthly  average  of  9.8  million 
payments  to  DoD  personnel,  1.2  million  commercial  invoices,  600,000  travel 
vouchers  or  settlements,  500,000  savings  bonds,  and  120,000  transportation 
bills  of  lading.  The  agency’s  monthly  disbursements  totaled  approximately 
$24  billion. 

For  year  2000  (Y2K)  purposes,  DFAS  identified  45‘  systems  as  mission  critical. 
For  purposes  of  testing  mission-critical  systems  for  Y2K  compliance,  DFAS 
identified  its  business  processes  considered  mission  critical  and  developed  plans 
to  test  those  business  processes.  Mission-critical  processes  are  defined  as  fliose 
that,  if  not  performed,  would  preclude  or  immediately  impair  the  chsbursal,  pay, 
and  accounting  functions.  Specifically,  DFAS  identified  the  following  seven 
mission-critical  business  processes:  civilian  pay,  nulitary  pay,  conhactor  and 
vendor  pay,  transportation  pay,  travel  pay,  accounting,  and  disbursing. 

End-to-End  Testing.  The  “end-to-end”  process  is  the  flow  of  data  through  a 
set  of  interconnected  systems  that  performs  a  core  busings  process,  function,  or 
mission.  Data  flow  begins  with  the  initial  input  of  data  into  the  first  system  and 
ends  with  the  final  receipt  of  data  in  the  last  system  and  receipt  of  output  by  the 
user.  The  purpose  of  Y2K  end-to-end  testing  is  to  verify  that  the  set  of 
interrelated  systems  supporting  DFAS  business  processes,  such  as  DoD 
transportation  pay,  operates  and  appropriately  processes  Y2K-related  data. 

DFAS  has  organized  its  end-to-end  testing  into  seven  testing  “events,”  one  for 
each  critical  mission  or  business  process.  DFAS  has  further  divided  each  event 
or  business  process  into  “threads.  ”  A  thread  represents  an  end-to-end  trace  of 
data  using  a  minimally  representative  sample  of  input  data  through  a  system  or 
set  of  systems  to  produce  a  sample  of  output  data.  Each  event  can  contain  one 
or  more  “threads”  that  track  to  a  critical  business  process. 

DFAS  Y2K  End-to-End  Master  Plan.  DFAS  issued  the  “DFAS  Y2K  End-to- 
End  Master  Plan,”  revision  2.3,  on  June  21, 1999  (the  DFAS  Master  Plan), 
specifically  for  accomplishing  Y2K-related  end-to-end  testing  of  its  mission- 
critical  business  processes.  The  DFAS  Master  Plan  identifies  roles  and 
responsibilities;  assumptions  and  constraints  related  to  testing;  interfaces  with 
non-DFAS  organizations;  and  requirements  for  planning,  testing,  and  reporting 
on  test  results. 

Roles  and  Responsibilities.  DFAS  designated  a  Y2K  project  manager 
and  functional  proponent  at  DFAS  headquarters  with  overall  Y2K  testing 


'  We  identified  42  DFAS  mission-critical  systems  in  previous  audit  reports.  DFAS  recently  added  three 
systems  to  its  mission-critical  list:  the  Standard  Accounting  and  Reporting  System  -  One  Bill  Pay,  the 
Standard  Army  Financial  Accounting  and  Reporting  System,  and  the  Standard  Base  Supply  System. 
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responsibility.  DFAS  allowed  its  functional  proponents  to  assign  event  leaders 
and  thread  leaders  to  execute  the  end-to-end  testing.  Also,  DFAS  assigned  roles 
and  responsibilities  to  system  managers  for  controlling  their  segment  of  the  end- 
to-end  testing  process. 

Assumptions  and  Constraints.  Because  of  limited  time  and  resources, 
the  DFAS  Master  Plan  acknowledged  constraints  and  identified  assumptions 
related  to  Y2K  end-to-end  testing.  For  example,  they  included  the  assumption 
that  third-party  software  and  computing  platforms  are  y2K  compliant,  that 
operations  and  compliance  testing  takes  precedence  over  end-to-end  testing,  that 
partner  organizations  will  conduct  their  own  internal  end-to-end  tests  and 
provide  input  for  DFAS,  or  that  all  mission-critical  systems  will  have 
contingency  plans  in  place. 

Interface  Requirements.  The  DFAS  Master  Plan  states  that  each  test 
event  will  include  critical  automated  interfaces  with  other  departments  and 
agencies.  However,  because  of  sizing  limitations  within  the  DoD  megacenters 
that  support  testing,  DFAS  and  the  other  DoD  organizations  may  not  be  able  to 
run  true  end-to-end  tests  simultaneously.  Rather,  each  organization  would 
mainfain  sufficient  control  of  its  segment  of  the  end-to-end  testing  process  to 
ensure  the  integrity  of  the  data  flow  from  one  system  to  the  other. 

Planning,  Testing,  and  Reporting.  The  DFAS  Master  Plan  specified 
requirements  for  the  following. 

•  Live  Versus  Simulation.  DFAS  planned  to  test  its  business 
processes  under  normal  operating  conditions  when  possible. 
Otherwise,  DFAS  was  to  use  a  “time  machine”  or  simulated 
operating  environment  and  document  the  reasons  and  the  associated 
risks. ^ 

•  Critical  Dates.  Although  the  DFAS  Master  Plan  did  not  designate 
specific  dates  for  testing,  it  did  recommend  that  testing  cover  the 
following  five  time  periods:  the  fiscal  year  2000  crossover,  calendar 
year  2000  crossover,  fiscal  year  2001  crossover,  calendar  year  2001 
crossover,  and  leap  year  (February  29,  2000).  DFAS  recommended 
that  dates  tested  be  consistent  with  dates  being  tested  by  interfacing 
systems. 

•  Baselines.  The  DFAS  Master  Plan  states  that  after  testing  the  dates, 
DFAS  organizations  will  compare  their  test  results  to  outcomes 
previously  determined  as  the  baseline.  (The  baseline  is  the  set  of 
known  end-to-end  test  inputs  and  outputs  extracted  from  systems  that 
have  been  certified  as  Y2K  compliant.)  Each  DFAS  organization 


time  machine  test  strategy  involves  setting  system  clocks  to  the  year  2000  and  operating  under  testing 
conditions.  Simulation  is  a  program  that  allows  testers  to  simulate  changing  dates  rather  than  actually 
changing  the  dates  during  normal  operations. 
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will  document  the  discrepancies  between  each  of  the  tests  and  the 
baseline. 

•  Data  Analysis  and  Documentation.  The  DFAS  Master  PIm 
requires  that  each  DFAS  organization  develop  and  document  in  its 
test  plan  a  data  collection  and  analysis  strategy  that  provides 
sufficient  information  to  support  end-to-end  test  design,  results,  and 
analysis.  The  DFAS  Master  Plan  leaves  the  details  for  data  analysis 
or  documentation  to  the  organizations  responsible  for  testing. 

Preliminary  Assessment.  On  May  28, 1999,  we  provided  DFAS  with 
preliminary  audit  results  in  a  memorandum  regarding  DFAS  planning  for  end- 
to-end  testing  in  general.  On  June  8,  1999,  the  Director  for  Information  and 
Technology,  DFAS,  responded  to  the  issues  of  roles  and  responsibilities.  Master 
Plan  checklists,  interfacing  systems,  critical  crossover  dates,  ^ta  collection  and 
analysis,  and  the  transportation  pay  event.  Appendix  B  contains  a  copy  of  our 
memorandum,  and  Appendix  C  is  the  DFAS  response. 

Defense  Transportation  Payment  System 


The  Defense  Transportation  Payment  System  (DTRS)  is  an  interactive,  real-time 
system  operating  in  the  DFAS  hidianapolis  Center.  DTRS  s^ndardues 
transportation  payment  processing  for  DoD.  The  DFAS  Indianapolis  Center 
currently  processes  aimual  payments  of  approximately  $1.7  billion  to  the 
transportation  industry  for  Army,  Air  Force,  and  other  DoD  organizations. 
DTRS  interfaces  include  traffic  management  systems,  DoD  finance  and 
accounting  systems,  the  General  Services  Administration  post-payment  audit 
system,  as  well  as  the  transportation  carrier  industry  systems. 

This  report  addresses  plans  for  conducting  Y2K  end-to-end  testing  for  DoD 
transportation  pay  and  those  test  results.  Other  reports  will  address  other  DFAS 
critical  business  processes. 

Objective 


The  overall  audit  objective  was  to  evaluate  the  effectiveness  of  the  planned  end- 
to-end  testing  in  the  DoD  transportation  pay  area.  Specifically,  we  reviewed  the 
DFAS  Master  Plan  and  the  plans  for  conducting  end-to-end  testing  within  the 
transportation  pay  process. 

See  Appendix  A  for  a  discussion  of  the  audit  scope  and  methodology,  the 
management  control  program,  and  prior  audit  coverage  related  to  Ae  audit 
objective.  We  will  review  the  sufficiency  of  contingency  plans  during  future 
audits. 
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Planning  and  Test  Results  for  Defense 
Transportation  Pay  End-to-End  Testing 

The  Defense  Finance  and  Accounting  Service  (DFAS)  made  significant 
improvements  in  its  end-to-end  testing  for  the  transportation  pay  area. 
The  DFAS  plan  was  to  use  the  system-level  test  results  to  fulfill  the 
requirements  of  end-to-end  tests.  We  questioned  that  rationale,  and 
DFAS  took  immediate  action  to  have  the  Joint  Interoperability  Test 
Command  review  the  system-level  tests  to  see  whether  they  met  the 
intent  of  end-to-end  tests  and  to  successfully  re-test  interfaces  between 
systems  that  were  uncertified  at  the  time  of  the  system-level  tests.  In 
addition,  DFAS  scheduled  a  test  with  the  Federal  Reserve  Bank  on 
September  15,  1999,  to  complete  the  end-to-end  testing  process.  As  a 
result,  DFAS  reduced  the  risks  associated  with  processing  transportation 
pay  entitlements  and  demonstrated  that  the  transportation  pay  systems 
should  continue  to  operate  in  the  year  2000, 

DoD  Transportation  Pay  Process 


Transportation  Pay  Business  Process.  The  DFAS  Indianapolis  Center 
supports  the  business  process  of  producing  transportation  payments  for  DoD, 
except  for  the  Navy  and  die  Marine  Corps^.  Two  automated  systems— the 
DTRS  and  the  Defense  Transportation  Payment  System-Accounting 
(DTRS-A)— support  the  transportation  pay  business  process.  The  two  systems 
interface  with  and  process  data  from  the  General  Services  Administration  and 
the  Military  Traffic  Management  Command.  Also,  the  two  systems  receive 
disbursing  support  from  the  Standard  Finance  System,  Redesign,  1  (SRD-1). 

Defense  Transportation  Payment  System.  DTRS  is  an  interactive,  real-time 
system  operating  in  the  DFAS  Indianapolis  Center  that  standardizes 
transportation  payment  processing  for  DoD.  The  DFAS  Indmapolis  Center 
currently  processes  annual  payments  of  approximately  $1.7  billion  to  the 
transportation  industry  for  Army,  Air  Force,  and  other  DoD  organizations. 
DTRS  interfaces  include  traffic  management  systems,  DoD  finance  and 
accounting  systems,  the  General  Services  Administration  post-payment  audit 
system,  and  the  transportation  carrier  industry  systems.  The  following  figure 
illustrates  the  DTRS  electronic  interfaces. 


’  Both  the  Navy  and  the  Marine  Corps  have  systems  that  interface  directly  with  the  Military  Traffic 
Management  Command  -  Financial  Management  System. 
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DTRS  ELECTRONIC  INTERFACES 


DFAS-IN  is  DFAS  Indianapolis  Center. 

EDI  is  electronic  data  interchange. 

GSA  is  the  General  Services  Administration. 

MTMC  is  the  Military  Traffic  Management  Command. 

Source;  DFAS  website:  http;//dfas4dod.dfas.mil/systems/DTRS/DTRS_EI.htm 


Transportation  Pay  Event  Test  Plan 

The  DFAS  Y2K  End-to-End  Master  Plan  (the  DFAS  Master  Plan)  initially 
identified  only  the  mission  of  the  transportation  pay  event  but  did  not  identify  a 
test  plan.  The  absence  of  a  test  plan  was  brought  to  the  attention  of  DFAS  in  a 
draft  report  issued  by  ManTech  Y2K  Solutions,  Incorporated  (a  subcontractor 
for  the  Joint  Interoperability  Test  Command  [JITC])'^.  In  addition,  our 
memorandum  dated  May  28,  1999,  addressed  the  need  foi  DFAS  to  develop  a 
test  plan  for  the  transportation  pay  event.  We  also  questioned  the  adequacy  of 
system-level  tests  to  suffice  for  end-to-end  tests.  (See  the  memorandum  in 
Appendix  B.) 


*  JITC  is  the  DoD  facility  for  the  test  and  evaluation  of  information,  communication,  and  intelligence 
systems  interoperability. 
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DFAS  responded  to  our  memorandum  by  stating  that  JITC  would  verify  that  the 
systems  defined  in  the  transportation  pay  event  had  conducted  system-level  tests 
that  met  the  requirements  of  end-to-end  testing.  DFAS  also  stated  that  if  any  of 
the  system-level  tests  were  not  approved  by  JITC,  steps  would  be  taken  to 
complete  all  or  any  portion  of  the  end-to-end  process  as  necessary 
(Appendix  C). 

The  DFAS  Master  Plan  states  that  assurance  can  be  obtained  through  use  of  test 
scenarios  for  end-to-end  testing  of  the  Defense  transportation  payment  business 
process.  The  end-to-end  testing  would  involve  live-testing  of  data  through 
DTRS,  DTRS-A,  and  all  interfaces  with  partnering  systems. 

We  issued  another  memorandum  to  the  Director  of  DFAS  on  July  9,  1999, 
expressing  concern  over  the  use  of  system-level  tests  as  end-to-end  tests. 

Because  the  system-level  testing  was  completed  on  March  2,  1999,  and  SRD-l 
was  not  certified  as  being  compliant  until  June  1999,  we  questioned  the  validity 
of  using  the  system-level  test  to  satisfy  the  requirements  of  end-to-end  testing. 

Test  Results  of  System-Level  Tests.  During  the  period  of  July  14  through  16, 
1999,  JITC  reviewed  the  results  of  the  system-level  tests  for  DTRS  and 
DTRS-A.  Based  on  its  initial  review,  JITC  determined  that  the  system-level 
testing  met  the  requirements  of  end-to-end  testing.  In  addition,  JITC  personnel 
agreed  with  our  concerns  about  the  SRD-l  system  not  beirig  compliant  at  the 
time  of  testing,  and  recommended  that  DFAS  conduct  additional  testing  on  the 
interface  between  DTRS-A  and  SRD-l. 

As  a  result  of  the  concern  that  we  and  JITC  had  about  the  interface  testing  being 
completed  before  the  SRD-l  being  certified  as  Y2K  compliant,  DFAS  scheduled 
a  retest  of  the  interface  between  the  DTRS  and  SRD-l.  DFAS  informed  us  that 
the  retest  was  successfiilly  completed  the  week  of  August  30,  1999.  The 
successful  completion  of  the  interface  test  demonstrated  that  the  transportation 
pay  process  should  continue  to  operate  as  intended  in  the  year  2000.  Because 
the  transportation  pay  process  electronically  pays  approximately  $1.7  billion 
annually  to  the  transportation  industry,  the  successfiil  re-test  of  the  interfaces 
also  indicates  significant  improvement  in  reducing  the  risks  associated  with 
processing  transportation  pay  entitlements.  In  addition,  DFAS  successfully 
completed  a  test  with  the  Federal  Reserve  Bank  on  September  15,  1999,  to 
complete  the  end-to-end  testing  process. 
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Appendix  A.  Audit  Process 


This  report  is  one  in  a  series  of  reports  being  issued  by  the  Inspector  General, 
DoD,  in  accordance  with  an  informal  partnership  with  the  Chief  Information 
Officer,  DoD,  to  monitor  DFAS  efforts  to  address  the  Y2K  computing 
challenge.  For  a  list  of  audit  projects  addressing  the  issue,  see  the  Y2K  web 
page  on  IGnet  at  http://www.ignet.gov. 

Scope 

We  reviewed  Y2K  reporting  requirements  and  policies  that  the  Office  of  the 
Secretary  of  Defense  and  DFAS  issued.  We  reviewed  the  DFAS  Master  Plan 
and  held  discussions  with  DFAS  managers  and  JITC  evaluators  to  obtain 
additional  information  and  clarification  on  the  transportation  pay  event  and  the 
test  results  of  the  system-level  tests  qualifying  as  end-to-end  tests.  At  the  time 
of  this  au^t,  the  DFAS  Indianapolis  Center  processed  annual  payments  of 
approximately  $1.7  billion  to  the  transportation  industry  for  Army,  Air  Force, 
and  other  DoD  organizations. 

DoD- Wide  Corporate-Level  Government  Performance  and  Results  Act 
Goals.  In  response  to  the  Government  Performance  and  Results  Act,  the 
Department  of  Defense  has  established  2  DoD-wide  goals  with  7  subordinate 
performance  goals.  This  report  pertains  to  achievement  of  the  following 
objectives  and  goal. 

DOD-wide  Goal:  Prepare  now  for  an  uncertain  future  by  pursuing  a 
focused  modernization  effort  that  maintains  U.S.  qualitative 
superiority  in  key  warfighting  capabilities.  Transform  the  force  by 
exploiting  the  Revolution  in  Military  Affairs,  and  reengineer  the 
Department  to  achieve  a  21**  century  infrastructure.  Subordinate 
Performance  Goal:  Transform  U.S.  militay  forces  for  the  future. 
(DoD-2.2) 

DoD  Functional  Area  Reform  Goals.  Most  major  DoD  fimctional  areas  have 
also  established  performance  improvement  reform  objectives  and  goals.  This 
report  pertains  to  achievement  of  the  following  functional  area  objectives  and 
goals. 

•  Information  Technology  Management  Functional  Area. 
Objective:  Become  a  mission  partner.  Goal:  Serve  mission 
information  users  as  customers.  (Information  Technology 
Management-1 .2) 

•  Information  Technology  Management  Functional  Area. 
Objective:  Provide  services  that  satisfy  customer  information  needs. 
Goal:  Modernize  and  integrate  Defense  information  infrastructure. 
(Information  Technology  Management-2.2) 
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•  Information  Technology  Management  Functional  Area. 

Objective:  Provide  services  that  satisfy  customer  information  needs. 
Goal:  Upgrade  technology  base.  (Information  Technology 
Management-2 .3) 


•  Information  Technology  Management  Functional  Area. 

Objective:  Reengineer  DoD  business  practices.  Goal:  Modify 
existing  systems  and  monitor  new  systems  to  be  Y2K  compliant. 
(Financial  Management-4.3) 

General  Accounting  Office  High-Risk  Area.  The  General  Accounting  Office 
has  identified  several  high-risk  areas  in  DoD.  This  report  provides  coverage  of 
the  Information  Management  and  Technology  high-risk  area.  In  its 
identification  of  risk  areas,  the  Y2K  problem  has  been  specifically  designated  as 
high  risk  by  the  General  Accounting  Office. 

Methodology 

Audit  Type,  Dates,  and  Standards.  We  performed  this  program  audit  from 
April  through  September  1999,  in  accordance  with  auditing  standards  issued  by 
the  Comptroller  General  of  the  United  States,  as  implemented  by  the  Inspector 
General,  DoD.  We  did  not  use  statistical  sampling  methods  or  computer- 
processed  data  for  this  audit. 

Use  of  Technical  Assistance.  We  met  with  technical  experts  from  JITC  to 
discuss  their  views  on  the  sufficiency  of  the  system-level  tests  qualifying  as  end- 
to-end  tests,  and  we  met  with  software  engineers  from  the  Technical  Assessment 
Division  of  the  Analysis,  Planning,  and  Quality  Assurance  Division.  JITC  is 
the  DoD  facility  for  the  test  and  evaluation  of  information,  communication,  and 
intelligence  systems  interoperability.  JITC  also  has  experienced  test 
professionals  available  on  a  fee-for-service  basis.  The  JITC  mission  is  to  be  the 
preeminent  information  systems  evaluator,  reducing  the  risk  to  the  warfighter  by 
providing  value-added  process  and  product  evaluations,  operational  assessments, 
and  technical  assistance  thoughout  die  life  cycle  of  DoD  systems. 

Contacts  During  the  Audit.  We  visited  or  contacted  individuals  and 
organizations  within  DoD.  Further  details  are  available  upon  request. 

Management  Control  Program.  We  did  not  review  the  management  control 
program  related  to  the  overall  audit  objective  because  DoD  recognized  the  Y2K 
issue  as  a  material  management  control  weakness  area  in  the  FY  1998  Annual 
Statement  of  Assurance. 

Summary  of  Prior  Coverage 

The  General  Accounting  Office  and  the  Inspector  General,  DoD,  have 
conducted  multiple  reviews  related  to  the  Y2K  issues.  General  Accounting 
Office  reports  can  be  accessed  over  the  Internet  at  http://www.gao.gov/. 
Inspector  General,  DoD,  reports  can  be  accessed  over  the  Internet  at 
http://www.dodig.osd.mil/. 
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Appendix  B.  Inspector  General,  DoD,  Interim 
Reporting  on  End-to-End  Test  Plans 


INSPECTOR  GENERAL 

DEPARTTMENT  OF  DEFENSE 
400  ARMY  NAVY  DRIVE 
ARUNQTON,  VIRGINIA  22202-2884 


FOR  DIRECTOR  FOR  INFORMATION  AND  TECHNOLOGY, 
DEFENSE  FINANCE  AND  ACCOUNTING  SERVICE 


SUBJECT:  Status  of  Audits  of  Finance  Functional  Area  Year  2000  End-to-End  Tests 


In  April  1999,  we  initiated  tbe  following  audit  projects  to  evaluate  the 
effectiveness  of  Y2K  end-to-end  testing  by  the  Defense  Finance  and  Accounting 
Services  PFAS).  The  projects  were  aligned  by  functional  area  in  accordance  with  the 
functional  break-out  identified  by  DFAS 


Civilian  Pay 

Military/Rctirec/Annuitanl  Pay 

Vendor/Contractor  Pay 

Transportation  Pay 

Disbursing 

Accounting 

Travel  Pay 


Project  9FG-9025 
Project  9FG-9026 
Projca9FG-9027 
Project  9FG-9028 
Project  9FG-9029 
Project  9FG-9030 
Project  9FG-9031 


Our  review,  to  date,  has  focused  on  analyzing  the  adequacy  of  test  plans  for  the 
seven  areas.  We  evaluated  the  adequacy  of  the  DFAS  Y2K  test  plans  using 
requirements  contained  in  the  DoD  Y2K  Management  Plan,  Version  2.1,  Appendix  I; 
the  DFAS  Y2K  Management  Plan,  Version  1.0;  the  DFAS  Y2K  End-to-End  Master 
Plan,  Version  2. 1;  the  DFAS  Regulation  8000. 1-R,  "Information  Management  and 
Instruction  Guidance,"  Version  5.0;  and  the  GAO  Operational  Evaluation  Assessment 
Tool.  We  anticipate  future  audits  will  assess  test  results  and  contingency  planning 
efforts  by  DFAS. 


Because  of  the  urgency  of  Ywr  2000  efforts,  our  intent  is  to  communicate 
potential  areas  of  concern  as  quickly  as  possible  so  that  management  may  address  these 
issues  in  a  timely  manner.  The  attachment  to  this  memorandum  reports  the  initial 
results  of  our  review.  During  our  preliminary  review,  we  identified  concerns  regarding 
the  adequacy  of  DFAS  planning  efforts  for  functional  end-to-end  testing.  If  these 
concerns  are  not  addressed,  there  is  increased  risk  that  DFAS  end-to-end  testing  may 
not  detect  a  significant  Y2K  problem.  We  may  include  these  and  any  additional  issues 
in  a  draft  report  at  a  later  date.  We  request  that  you  provide  a  response  to  this 
memorandum  by  June  8,1999.  If  there  are  any  questions,  please  contact  Ms.  Kimberley 
Caprio,  Program  Director  at  (703)  604-9139  or  DSN  664-9139, 


F.  Jay  Lane 
Director 


Finance  and  Accounting  Directorate 
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DFAS  has  made  significant  progress  in  addressing  testing  requirements  for  its 
functional  areas  including  the  issuance  of  a  Master  Plan,  identification  of  levels  of 
responsibility,  and  checklists  for  lest  planning  purposes.  During  our  review,  we 
identified  the  following  concerns  that  should  be  addressed  by  DFAS.  On  May  27, 

1999,  we  met  with  DFAS  officials  to  discuss  the  concerns  and  actions  to  be  taken. 

1.  Roles  and  Responsibilities-  The  DFAS  Master  Plan  identified  four  levels  of 
responsibility  for  end  to  end  testing  including  Headquarters  functional  proponents, 
systems  managers,  event  leaders,  and  thread  leaders.  The  Plan  defined  roles  and 
responsibilities  for  functional  proponents  and  systems  managers,  but  did  not  provide 
details  on  the  responsibilities  for  either  the  event  or  thread  leaders.  During  fiie  May 
27, 1999  meeting,  DFAS  Headquarters  personnel,  acknowledged  the  need  for  oversight 
and  agreed  to  provide  the  details  immediately . 

In  addition,  the  Master  Plan  was  not  issued  until  May  11,  1999,  and  in  some  cases  had 
not  arrived  at  the  event  leader  level  until  May  18, 1^9.  However,  functional  event 
plans  and  allocation  of  responsibilities  was  already  occurring.  As  a  result,  the 
individuals  delegated  the  responsibilities  may  not  have  been  appropriate.  For  example, 
for  the  Travel  pay  event,  the  same  person  was  tasked  as  both  the  functional  proponent 
and  the  event  leader.  As  a  result,  it  precludes  the  separation  of  duties  by  allowing  one 
function  to  oversee  the  other  function.  To  ensure  that  the  4  levels  of  responsibility  are 
appropriately  staffed,  the  DFAS  Headquarters  Project  Office  should  review  the 
assigned  personnel  and  ensure  that  they  are  aware  and  understand  their  delegated 
responsibilities. 

2.  Master  Plan  Checklists.  The  DFAS  Y2K  Master  Plan  included  four  checklists 
to  be  used  by  DFAS  Headquarters  personnel,  the  functional  area  proponent,  the  event 
leader,  and  the  tester.  These  checklists  require  DFAS  personnel  to  assess  the 
effectiveness  of  the  end-to-end  testing  program  at  each  designated  level  including  such 
items  as  assessing  the  adequacy  of  testing  staff,  funds,  and  interface  agreements.  The 
DFAS  Master  Plan  stated  that  these  checklists  would  “provide  independent  auditors 
with  evidence  of  compliance  with  the  end-to-end  test  requirements,''  however,  the 
Master  Plan  did  not  make  completion  of  the  checklists  mandatory. 

We  believe  the  checklists  should  be  mandatory  and  maintained  at  the  fimctional  level 
along  with  test  results.  The  2  to  3  page  checldists  provide  an  excellent  means  to 
ensure  and  document  that  essential  steps  were  taken  prior  to  performing  end  to  end 
testing  of  DFAS  functional  areas.  Completion  of  the  planning  section  of  these 
checklists  provides  a  tool  to  help  ensure  compliance  with  the  Master  Plan  requirements 
and  allow  for  early  corrections  of  deviations  or  omissions  from  the  plan.  Further,  use 
of  the  checklist  affords  standardization  of  the  process  used  throughout  DFAS  for  end 
to  end  test  planning  efforts.  Without  the  use  of  the  checklists,  DFAS  lacks  assurance 
that  the  testing  was  con^Iete,  adequate,  and  consistent.  We  also  believe  that  a 
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signature  block  or  notation  should  be  included  in  the  checklists  to  establish 
accountability  for  the  responses  and  to  facilitate  quick  actions  should  a  problem  arise 
later. 

3,  Interfacing  Systems,  DFAS  relies  heavily  on  interfacing  systems  to  provide 
the  majority  of  data  included  in  DFAS  systems.  As  such,  coordination  and 
compatibility  of  data  exchanged  with  interfacing  systems  is  critical  to  ensuring 
successfiil  Y2K  end  to  end  tests.  If  data  from  a  non-compliant  system  feeds  into  a 
DFAS  system,  the  potential  exists  for  the  DFAS  system  to  not  be  able  to  function 
properly  after  y2K.  The  level  of  assurance  being  obtained  by  DFAS  ftmcdonaj  area 
officials  regarding  Y2K  compliance  of  interfacing  systems  varies  from  exchanging 
documentation  to  merely  assuming  that  interfacing  systems  are  compliant  or  relying  on 
verbal  responses. 

Given  the  significant  potential  impact  of  interfaces  on  successful  testing,  we  believe 
that  DFAS  functional  leaders  should  take  the  extra  step  to  validate  that  key  interfacing 
systems  are,  in  fact,  compliant.  Information  on  the  compliance  of  each  DoD  mission 
critical  system  should  be  available  in  the  OSD  database.  As  such^  DFAS  personnel  for 
the  functional  areas  should  be  able  to  access  the  database  and  validate  that  those 
applicable  interfacing  partners  are  clearly  designated  as  Y2K  compliant  before  entering 
the  end-to-end  test.  We  discussed  this  matter  with  DFAS  Headquarters  officials  who 
agreed  that,  while  they  are  only  testing  with  compliant  interfacing  partners,  it  is 
reasonable  that  DFAS  review  the  database  to  ensure  that  interfacing  partners  are 
compliant  prior  to  testing. 

4.  Critical  Crossover  Dates.  The  DFAS  Master  Plan  identified  5  critical  cross¬ 
over  dates  as  mandatory  for  inclusion  during  end-to-end  tests.  The  dates  are  consistent 
with  the  5  dates  recommended  by  the  Assistant  Secretary  Defense  (Cor^nd,  Control, 
Communications,  and  Intelligence).  The  dates  are  fiscal  year  2000,  calendar  year 

2000,  leap  year  crossing  (February  29, 2000),  fiscal  year  2001,  and  calendar  year 

2001.  Developers  of  DFAS  test  plans  have  not  planned  to  test  all  5  dates.  For 
example,  the  Marine  Cotp  Total  Force  System  is  only  testing  the  leap  year  2000 
crossover.  The  Computerized  Accounts  Payable  System'  is  not  testing  the  fiscal  year 
and  the  calendar  year  2001  crossovers.  The  reduced  number  of  dates  being  tested  is  a 
result  of: 

•  The  test  plans  being  developed  prior  to  the  issuance  of  the  DFAS  Master  Plan  on 
May  11. 1999, 

•  Personnel  pay  systems,  for  example,  not  being  impacted  by  fiscal  year  changes, 

•  Funding  being  allocated  based  on  test  plans  developed  prior  to  the  Master  Plan. 

In  order  to  ensure  compatibility  of  interfacing  systems,  it  is  important  that  the  same 
dates  arc  tested,  particularly  where  DFAS  systems  feed  data  to  other  systems.  For 
example,  data  from  systems  within  the  Travel  Pay  test  event  feed  into  systems  within 
the  Disbursing  test  event.  Further,  once  processed  within  disbursing,  data  is  fed  to  both 
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accounting  and  back  to  travel  systems.  Incoming  files  to  the  Automated  Disbursing 
System  (ADS)  (part  of  the  Disbursing  test  event)  plan  to  test  all  dates  specified  in  the 
DFAS  Y2K  Master  Plan,  However,  the  Travel  Pay  test  event  does  not  plan  to  test  the 
fiscal  year  2000  to  2001  crossover.  As  a  result,  the  potential  exists  that  data  relying  on 
the  fiscal  year  2000  to  2001  crossover  may  not  function  properly.  Meanwhile,  DFAS 
may  report  a  successful  test  based  on  the  less  than  5  dates  being  tested. 

It  is  important  that  interfacing  systems  select  test  dates  in  a  similar  manner  to  ensure 
Y2K  data  flows  through  each  system  appropriately.  DFAS  functional  leaders  should 
ensure  that  critical  crossover  dates  for  each  of  the  seven  functional  areas  axe  compatible 
prior  to  testing. 

5.  Data  Collection  and  Analysis.  The  DoD  Y2K  Management  Plan  states  that 
Y2K  event  output  products  such  as  plans  and  procedures  should  specify  in  detail  what 
data  needs  to  be  collected,  who  will  analyze  the  data,  and  how  it  will  be  analyzed. 
Essentially,  the  requirement  is  to  define  expected  test  results.  Consistent  with  the  DoD 
Plan,  the  DFAS  Master  Plan  requires,  as  exit  criteria  to  the  test  planning  phase,  that 
responsible  parties  specify  pass/fail  criteria  for  all  tests,  that  data  collection  procedures 
are  in  place,  and  mechanisms  needed  to  capture  data  are  installed.  The  DFAS  Master 
Plan,  however,  does  not  specify: 

•  What  types  of  data  should  be  collected  to  ensure  consistency  in  reporting  test 
results. 

•  A  methodology  for  each  DFAS  organization  to  document  the  data  collection 
process  in  the  appropriate  Event  Plan 

For  the  7  DFAS  functional  events,  data  collection  and  data  analysis  plans  arc 
either  nonexistent  or  do  not  ensure  the  tests  will  be  judged  objectively.  For  example, 
the  Defense  Industrial  Financial  Management  System  (DIFMS)  Test  Plan,  which  is  part 
of  the  Accounting  Test  Event,  plans  to  review  reports,  queried  data,  and  DIFMS 
screens  to  accomplish  data  analysis,  but  did  not  establish  expected  test  results  criteria  or 
a  baseline  that  could  be  used  to  determine  the  adequacy  or  accuracy  of  the  reports , 
queries,  and  screens.  As  another  example,  the  Civilian  Pay  Event  lacks  either  a  data 
collection  plan  or  a  data  analysis  plan.  Instead,  the  Event  Leader  indicated  that  years  of 
prior  testing  and  DCPS  experience  will  identify  discrepancies  should  they  arise. 

Both  DoD  and  DFAS  require  the  establishment  of  a  structured  approach  to  testing 
including  identifying  expected  outcomes,  test  participants,  and  other  details.  Without 
such  plans,  there  is  no  organized  or  standardized  approach  between  the  participating 
systems,  nor  any  assurance  that  test  goals  arc  met  and  tests  were  successful.  Given  the 
nature  of  end-to-end  testing,  with  its  large  numbers  of  participating  or  “partner’' 
systems,  it  is  prudent  to  ensure  that  the  data  collection  is  as  consistent  as  possible  for 
each  event,  and  that  the  analysis  of  the  test  data  is  objective.  Without  the  definition  of 
data  collection  and  data  analysis  plans  before  testing  begins,  this  will  be  difficult. 
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DFAS  Rinctional  Test  Leaders  need  to  ensure  that  detailed  test  collection,  results,  and 
analysis  requirements  are  clearly  defined  prior  to  testing. 

6,  Pav  Event.  DFAS  identified  Transportation  Pay  as  one  of  the 

7  functional  areas  for  testing  purposes.  However,  DFAS  has  not  yet  developed  an  end- 
to-end  test  plan  for  the  went.  There  are  two  systems  Involved  in  transportation,  the 
Defense  Transportation  Pay  System  (DTRS)  and  the  Military  Traffic  Management 
Command  -  Financial  Management  System  (MTMC-FMS).  The  Transportation  Pay 
Event  Leader  stated  that  MTMC-FMS  testing  during  Y2K  conversion  process 
accomplished  the  end-to-end  requirements  of  the  Master  Plan.  DFAS  has  subsequently 
contracted  with  the  Joint  Interoperability  Testing  Command  (JITC)  to  independently 
verify  and  validate  the  prior  testing.  We  plan  to  follow-up  on  fiiis  functional  area 
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Appendix  C.  Defense  Finance  and  Accounting 
Service  Response  to  Inspector  General,  DoD, 
Interim  Report 


DFAS-HQ/S 


DEFENSE  FINANCE  AND  ACCOUNTING  SERVICE 


1931  JEFFERSON  DAVIS  HIGHWAY 
ARLINGTON,  VA  22240-S291 

JUN  '  8  1999 


MEMORANDUM  FOR  DIRECTOR,  FINANCE  AND  ACCOUNTING  DIRECTORATE 

OFFICE  OF  THE  INSPECTOR  GENERAL,  DEPARTMENT  OF 
DEFENSE 


SUBJECT:  Status  Of  Audits  of  Financial  Functional  Area 

Year  2000  End-to-End  Tests 

The  attached  outlines  Defense  Finance  and  Accounting 
Service  (DFAS)  response  to  the  DoD  Inspector  General's  (IG> 
initial  review  of  and  concerns  about  DFAS'  End-to-End  Test  (E2E) 
Plans.  DFAS  recognizes  that  a  great  deal  of  work  is  still  to  be 
done  to  ensure  all  necessary  requirements  for  E2E  are 
accomplished.  To  meet  this  goal,  DFAS  has  conducted  meetings 
with  event  and  thread  leaders  to  review  all  E2E  guidelines  and 
requirements , 

All  concerns  addressed  in  the  DoD  IG's  memo  are  being 
addressed. 

Roles  and  Responsibilities:  Concur.  Action  to  expand 
event  and  thread  leader  roles  will  be  accomplished  by 
June  25,  1999. 

Master  Plan  Checklist:  Non-concur.  DFAS  will  not  mandate 
the  checklist. 

Interfacing  Systems:  concur.  This  action  is  considered 
completed,  but  with  periodic  updates. 

Critical  Crossover  Dates;  Concur.  This  action  is 
completed. 

Data  Collection  and  Analysis:  Concur.  This  is  an  ongoing 
action  with  no  specific  target  date. 

Transportation  Pay  Event;  Concur.  This  is  an  ongoing 
action  with  a  target  completion  date  of  June  30,  1999. 
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Roles  and  Responsibilities.  Concur,  DFAS  acknowledges  that 
testing  and  planning  activities  were  occurring  prior  to  the 
issuance  of  the  Master  Plan  on  6  May.  However,  several 
coordination  meetings  had  already  occurred  and  guidance  on 
developing  Event  Plans  was  issued  on  31  March.  DFAS  also 
acknowledges  the  need  to  clarify  and  augment  the  roles  and 
responsibilities  of  the  event  and  thread  leaders  in  the 
DFAS  E2E  Master  Plan.  We  are  currently  making  site  visits 
and  meeting  with  the  testing  teams  to  clarify  roles  and 
responsibilities  and  are  updating  the  Master  plan  as  well. 
It  should  be  noted  there  may  be  an  overlap  in  the  area  of 
responsibility,  due  to  the  fact  that  the  internal  DFAS 
support  structure  for  each  business  process /application  has 
a  great  bearing  upon  the  specific  breakout  of  roles  and 
responsibilities.  DFAS  does  not  view  this  as  a  conflict  or 
an  inappropriate  assignment  of  duties. 

Master  Plan  Checklist.  Non-concur,  DFAS  designed  and 
issued  these  checklists  as  tools  to  assist  DFAS  personnel 
responsible  for  planning,  tracking,  and  conducting  end  to 
end  testing.  Because  each  business  area/application  has  a 
normal  testing  practice  already  established,  DFAS  did  not 
make  the  checklists  mandatory,  and  would  prefer  to  keep  the 
use  of  checklists  optional.  However,  DFAS  will  encourage 
the  use  of  the  checklists  whenever  possible. 

Interfacing  Systems.  Concur.  DFAS  agrees  that  coordination 
and  compatibility  of  data  exchange  between  DFAS  systems  and 
their  interface  partners  is  essential  to  a  successful  Y2K 
effort.  DFAS  has  pursued  this  goal  for  the  past  two  years, 
DFAS  has  established  Interface  Agreements  with  all  of  its 
interface  partners.  This  effort  generated  in  excess  of 
1400  agreements.  In  addition,  DFAS  has  tracked  and  updated 
on  a  monthly  basis  the  status  of  testing  and  compliance  of 
each  of  its  interface  partners.  DFAS  system  mangers  are 
well  aware  of  the  status  of  each  of  its  partners.  DFAS 
will  continue  to  track  ard  monitor  the  status  of  its 
interface  partners  mission  critical  and  other. 

Critical  Crossover  Dates.  Concur.  DFAS  acknowledges  the 
importance  of  testing  as  many  dates  as  possible,  and  the 
coordination  of  these  dates  among  partners.  All  DFAS 
managers  have  been  encouraged  to  coordinate  this  initiative 
with  all  pertinent  parties.  It  must  be  understood  that 
dates  do  not  necessarily  play  an  important  part  in  the 
relationship  of  one  system  to  another.  The  DFAS  E2E  Master 
Plan  has  recently  been  updated  to  empower  the  Functional 
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Managers  with  determining  which  dates  are  critical  for 
testing  within  their  specific  business  process.  We  have 
also  hired  JITC  to  independently  validate  and  verify  our 
planning  efforts. 

Data  Collection  and  Analysis.  Concur.  DFAS  agrees  that 
current  plans  lack  specific  exit  criteria  and  we  are  taking 
action  to  strengthen  this  area  of  our  plans.  DFAS  8000.1- 
R,  Part  E,  Chapter  3,  Test  and  Evaluation  provides  guidance 
concerning  data  collection  and  analysis.  Our  central 
design  activities  normally  plan  and  execute  their  tests, 
using  this  guidance,  precluding  the  need  for  specific 
guidance  to  be  issued  relative  to  E2E  testing. 

Each  testing  agent  within  DFAS  implements  the  regulation 
within  their  own  construct,  resulting  in  a  non-standard, 
but  successful,  data  collection  and  analysis  process. 
Because  Y2K  E2E  testing  requirements  are  not  system 
centric,  but  business  process  centric,  we  have  hired  JITC 
to  independently  validate  and  verify  our  E2E  planning  and 
testing  efforts.  The  JITC  analysis/evaluation  will 
document  specific  risks  associated  with  data  collection  and 
analysis  procedures,  in  sufficient  time  for  us  to  take 
corrective  action. 

Another  measure  of  risk  mitigation  is  to  conduct  site 
visits,  where  we  meet  with  all  thread  participants.  During 
these  meetings  we  are  emphasizing  the  need  for  adequate 
documentation  of  their  data  collection  and  analysis 
procedures,  as  well  as  documenting  version  control  and 
configuration  management  procedures. 

Transportation  Pay  Event.  Concur.  In  addition  to  the 
Transportation  Pay  System,  DFAS  has  identified  several 
other  systems,  which  claim  completion  of  the  end  to  end 
testing  initiative.  JITC  will  be  used  to  verify  that  these 
systems  indeed  have  met  E2E  requirements.  If  any  system 
fails  to  pass  the  validation  of  JITC,  steps  will  be 
initiated  to  complete  all  or  any  portion  of  the  E2E  process 
that  needs  to  be  completed. 
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Appendix  D.  Report  Distribution 


Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition  and  Technology 
Director,  Defense  Logistics  Studies  Information  Exchange 
Under  Secretary  of  Defense  (Comptroller) 

Deputy  Chief  Financial  Officer 
Deputy  Comptroller  (Program/Budget) 

Assistant  Secretary  of  Defense  (Command,  Control,  Communications,  and  Intelligence) 
Deputy  Chief  Information  Officer  and  Deputy  Assistant  Secretary  of  Defense  (Chief 
Information  Officer  Policy  and  Implementation) 

Principal  Director  for  Year  2000 


Joint  Staff 

Director,  Joint  Staff 

Department  of  the  Army 

Inspector  General,  Department  of  the  Army 
Auditor  General,  Department  of  the  Army 

Department  of  the  Navy 

Assistant  Secretary  of  the  Navy  (Financial  Management  and  Comptroller) 
Inspector  General,  Department  of  the  Navy 
Auditor  General,  Department  of  the  Navy 
Inspector  General,  Marine  Corps 

Department  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  (Financial  Management  and  Comptroller) 
Inspector  General,  Department  of  the  Air  Force 
Auditor  General,  Department  of  the  Air  Force 
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other  Defense  Organizations 

Director,  Defense  Contract  Audit  Agency 
Director,  Defense  Finance  and  Accounting  Service 
Director,  Defense  Information  Systems  Agency 

Inspector  General,  Defense  Information  Systems  Agency 
United  Kingdom  Liaison  Officer,  Defense  Information  Systems  Agency 
Director,  Defense  Logistics  Agency 
Director,  National  Security  Agency 

Inspector  General,  National  Security  Agency 
Inspector  General,  Defense  Intelligence  Agency 
Defense  Systems  Management  College 

Non-Defense  Federal  Organizations 

Office  of  Management  and  Budget 

Office  of  Information  and  Regulatory  Affairs 
General  Accounting  Office 

National  Security  and  International  Affairs  Division 
Technical  Information  Center 

Director,  Defense  Information  and  Financial  Management  Systems,  Accounting,  and 
Information  Management  Division,  General  Accounting  Office 

Congressional  Committees  and  Subcommittees,  Chairman  and 
Ranking  Minority  Member 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 

Senate  Committee  on  Armed  Services 

Senate  Committee  on  Governmental  Affairs 

Senate  Special  Committee  on  toe  Year  2000  Technology  Problem 

House  Committee  on  Appropriations 

House  Subcommittee  on  Defense,  Committee  on  Appropriations 
House  Committee  on  Armed  Services 
House  Committee  on  Government  Reform 

House  Subcommittee  on  Government  Management,  Information,  and  Technology, 
Committee  on  Government  Reform 

House  Subcommittee  on  National  Security,  Veterans  Affairs,  and  International 
Relations,  Committee  on  Government  Reform 
House  Subcommittee  on  Technology,  Committee  on  Science 
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Audit  Team  Members 

The  Finance  and  Accounting  Directorate,  Office  of  the  Assistant  Inspector 
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